Private network access point router for interconnecting among internet route providers

ABSTRACT

An improvement to the Private Network Access Point (PNAP) packet switched network described in U.S. Pat. No. 6,009,081, where two customers connected to the same PNAP will exchange traffic through the PNAP without transiting over the backbones of the Internet. In addition, a multi-homed customer connected to the PNAP is provided with access to the PNAP optimized routing table so that the customer will also have the ability to know the best route for a particular destination. In this way, if a multi-homed customer connected to the PNAP is directly connected to a particular NSP to which a destination is also connected, the PNAP customer can use the PNAP information regarding the NSP to send the information to the destination through that commonly connected NSP in the most direct fashion.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains generally to routing information packets in anetwork involving a plurality of traffic carrying networks, and moreparticularly to an improvement in routing described in U.S. Pat. No.6,009,081.

2. Description of the Background Art

The present invention is an improvement on the invention of improvementin routing described in U.S. Pat. No. 6,009,081, and assigned to theassignee hereof. Additional background information can be found theaforesaid patent, as well as in the book entitled “Internet RoutingArchitectures” by Bassam Halabi, New Riders Publishing, 1997, which ishereby incorporated herein by reference.

As indicated in U.S. Pat. No. 6,009,081, column 6, lines 62-66, a PNAPor “Private Network Access Point” can be thought of as being made up oftwo halves. One half connects to customers. The other half connects toNSPs or “National Service Providers”.

The Internet is a network of networks. A PNAP contains an ASimilaterthat determines the Internet interconnection matrix. ASimilater serversresiding within the PNAP collect and collate the routing data receivedfrom Network Service Providers (NSPs) to build a database of how theInternet is interconnected. The database shows the NSPs connected to thePNAP are interconnected as well as how they are connected to theircustomers. The PNAP receives each NSP's perspective of on Global RoutingTable which, when collated, includes identical routes from multipleNSPs, and that distillation of the sum of each NSP's view of the GlobalRouting Table is used to direct traffic from the customer to thedestination over the optimal path via another PNAP customer if availableor, otherwise, one of the NSP's connected to the PNAP.

BRIEF SUMMARY OF THE INVENTION

In accordance with an aspect of the invention, if two customers who areconnected to the same PNAP wish to communicate with each other, trafficwill be exchanged between those customers through the PNAP without evertransiting over the backbones of the NSPs.

According to another aspect of the invention, a multi-homed customerconnected to the PNAP is provided with access to the PNAP's optimizedversion of the Global Routing Table so that the customer will also havethe ability to know the best route for a particular destination.

According to a still further aspect of the invention, if a multi-homedcustomer connected to the PNAP is directly connected to a particular NSPto which a destination is also connected, the PNAP customer can, basedon information provided by the PNAP, send the information to thedestination through that commonly connected NSP.

According to another aspect of the invention, provision is made for therouting of traffic for customers who are multi-homed to multiple PNAPsin addition to one or more of the commonly connected NSPs.

A further aspect of the invention provides for routing traffic forcustomers who are not massively multi-homed, but are connected to morethan one PNAP.

Further objects and advantages of the invention will be brought out inthe following portions of the specification, wherein the detaileddescription is for the purpose of fully disclosing preferred embodimentsof the invention without placing limitations thereon.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1 is a schematic diagram shown two PNAPs with multi-homed customersaccording to an embodiment of the invention.

FIG. 2 and FIG. 3 are flow charts showing a method of the invention forcausing traffic between two customers of the same PNAP to be exchangedthrough the PNAP without transiting over the Internet.

FIG. 4 is a schematic diagram of a customer multi-home to a PNAP and aplurality of NSPs in accordance with the invention.

DESCRIPTION OF THE INVENTION

Referring more specifically to the drawings, for illustrative purposesthe present invention will now generally be described in connection withthe system configuration, setup and operational methodology shown inFIG. 1 through FIG. 3. It will be appreciated that the system may varyas to configuration, and that the method may vary as to the specificsteps and sequence, without departing from the basic inventive conceptsdisclosed herein.

Referring first to FIG. 1, in accordance with an embodiment of theinvention a first PNAP 20 and second PNAP 20 a are both shown as acircle with a vertical dashed line 21 dividing it in half. While morecustomers would typically be connected to the PNAPs 20, 20 a, the lefthalf of the PNAPs 20, 20 a are shown connected to two customers 1, 2 asan example to simplify the discussion. Furthermore, while two PNAPs 20,20 a are shown, there could be one or any other number of PNAPs. Whilecustomers 1, 2 are both shown connected to two PNAPs, a customer may beconnected only to one PNAP or any other number of PNAPs. Note also inthis regard, that the two customers are considered to be “multi-homed”because they are connected to more than one PNAP. In addition, customers1, 2 each have a second link that is connected to the Internet 22. Thisis also considered to be a “multi-homed” configuration. It will beappreciated, however, that it is not necessary for either customer 1 orcustomer 2 to be multi-homed to employ all of the inventive attributesdescribed herein. Also, while PNAP 20 will be referred to herein forsimplicity, the discussion is applicable to PNAP 20 a as well.

In the configuration shown, the right half of the PNAP 20 is connectedto a plurality of NSPs A, B, C, D, . . . N which, in turn, form theInternet 22 to which Internet users, such as destinations 3, 4 are alsoconnected. Note that the NSPs A-N do not exchange traffic amongthemselves through the PNAP 20. Traffic exchanges between NSPs A-N takesplace at public or private peering points (not shown).

The customers 1, 2 typically route their traffic through the PNAP 20from the left half to the right half. The PNAP 20 then selects the pathfrom the customers 1, 2 to the destinations 3, 4.

From U.S. Pat. No. 6,009,081, it will be understood that the PNAP 20contains an ASimilater that determines how everyone on the Internet 22is connected to everyone else. Hereinafter, the term “ASimilater” willbe used synonymously with the term “ASsimilator” in that patent. It willalso be understood that the Border Gateway Protocol, version 4 protocol(BGP4) used therein encompasses the concept of a “Global Routing Table”which may be defined as the list of all routes visible to each provider,both of its customers as well as its peers and their customers, ofeveryone to which they are connected. Briefly, an ASimilater serverinside the PNAP 20 receives a data “dump” of the Global Routing Tablefrom each of its NSPs A-N, and collates the data together to build adatabase of how the Internet 22 is interconnected. The database showshow all of the NSPs A-N are connected together as well as connections totheir customers. Once the ASimilater has populated this database, ituses the forward path or reverse path algorithm defined in U.S. Pat. No.6,009,081 to determine which routes are NSP A's customers, which are NSPB's customers, and so on, for all of the NSPs. In effect, the ASimilater“mines” this database. To summarize:

-   -   1. The ASimilater takes a dump of the Global Routing Table from        each NSP A-N.    -   2. The ASimilater collates the data from each NSP's perspective        of the Global Routing Table.    -   3. The ASimilater builds a summed Global Routing Table database        of the Internet 22's interconnection matrix.    -   4. The ASimilater determines which routes are NSP A's customers        and so on for all customers and for all other NSPs B-N. As a        clarification, note that each NSP is also sending routes of all        other NSPs to which it is connected.        The routing table inside the PNAP 20 also maps a plurality of        routes from customer 1 to customer 2 that go through the NSPs        A-N.

In accordance with the present invention, if neither customer 1 norcustomer 2 is multi-homed and those customers wish to communicate witheach other, traffic will be exchanged between those customers throughthe PNAP 20 without ever transiting over the backbones of the NSPs A-N.In the case of sending information from customer 1 to customer 2, therouting table inside the PNAP 20 would list the direct connection fromcustomer 1 to customer 2 through the left half of the PNAP 20 over thedotted path 25 as the optimum route. This means that communicationsbetween customer 1 and customer 2 who are connected to the PNAP 20 wouldalways use the dotted path 25 as the preferred path unless a failure orflaw prevents that path from being used, in which case traffic betweenthose customers would be exchanged through the Internet.

Accordingly, data packets would typically flow from customer 1 to thePNAP 20 and directly to customer 2 without traversing any of the NSPsA-N that comprise the Internet 22. This is illustrated in FIG. 2 andFIG. 3, which are method flow charts. In FIG. 2, the method begins atblock 30 and proceeds to block 31 which is the step of causing therouter within the PNAP to list the direct route through the PNAP as oneof its routes between two customers connected to the PNAP. The step ofthe next block 32 is causing the level of preference for the direct pathto be higher than for any other routes between the two customers. Thestep of the next block 33 is causing the router protocol to select thedirect route as being the best path between the two customers. Finally,the last block 34 of FIG. 2 is “end”. Similarly for FIG. 3, the methodbegins at block 36. The first step in block 37 is causing the customerrouter to forward a packet from customer 1 to customer 2 over the PNAPlink. The next step in block 38 is causing the PNAP router to forwardthe packet from customer 1 over the direct PNAP path to customer 2without transiting a service provider backbone. Finally the last block40 of FIG. 3 is “end”.

The potential for unacceptable path latency is reduced by this directconnection between customer 1 and customer 2. Path latency can, forexample, result from delay between the time when a device receives aframe and the time that frame is forwarded out the destination port, orthe delay caused by a shift to a more circuitous path due to an outage.

With regard to exchanging information between, for example, customers 1,2 and destinations 3, 4, usually there will be more than one route fromthe customers 1, to the destinations 3, 4. Therefore, the routers withinthe PNAP are used to forward packet traffic through the Internet 22 inan optimized fashion. The routers build routing tables that containtheir distillation of the summed Global Routing Table resulting in thebest paths to all the destinations from the PNAP's perspective. Theyboth advertise and receive route information to and from other routers.The routers keep track of next hop information that enables a datapacket to reach its destination. A router that does not have, a directphysical connection to the destination checks its routing table andforwards the packet to its next hop; that is, a router that it isdirectly connected to and is closer to that destination. This processrepeats until the traffic reaches its destination.

In a multi-homed configuration as shown in FIG. 1, if customer 1 wishesto send a packet to destination 3, it will see a link 23 to the Internet22 and a link 24 to the PNAP 20. As part of the BGP4 protocol, customer1 inherently has complete control over the outbound routing of itstraffic communications in this configuration. As such, said customer mayset the BGP4 local preference on the routes received by its router todestination 3 in order to cause it to prefer a particular link. Forexample, if destination 3 is connected to NSP D, customer 1 may set thepreferences within its router to prefer link 24 based on link 24 beingthe optimum link. Otherwise, link 23 to NSP D may be the preferred link.However, in the event that a fault or failure appears on the preferredlink, diversity considerations will cause the other link to be usedinstead.

In order for the customer to be able to set the preferences within itsrouter to cause it to prefer a particular link, the customer needsrouting information to know which path is optimum. Therefore, in amulti-homed configuration with the PNAP and another provider, thecustomer is given access to ASimilater data over its BGP feed to thePNAP. This is done so that the PNAP customer can effectively use boththeir PNAP and their other NSP pipe. Without the additional ASimilatordata in the form of BGP communities on the customers BGP feed from thePNAP, they are left with attempting to push traffic over the PNAP andprovider pipes in a sub-optimal fashion. Again, it may be preferred fora customer to use its pipe to NSP D for communicating with destinationsthat are connected to NSP D and to use the PNAP (and its externalconnections to NSPs A-N) for all other destinations. The optimized anddistilled Global Routing Table would be sent to the PNAP customer. Inthis example, the BGP4 attribute known as the “community” would be usedto tag NSP C customer routes as determined by ASimilater with the PNAPNSP C customer community. Since the customer has complete control overoutbound traffic, the customer can set the local preference in itsrouter to tag a particular route of multiple identical routes frommultiple sources as the preferred route. The higher the localpreference, the more preferred the route. For example, on the inboundpolicy applied to the routes received from the PNAP, any routes taggedwith the PNAP's community for NSP D could have their local preferenceset to 50 and every other route (not tagged) set to 150. On the BGP feedfrom NSP D, the customer could leave all routes at local preference 100which is the default. This allows the customer to optimize their routingso that the direct pipe to NSP is D is used for destinations on NSP Dand the PNAP 20 is used for other destinations, thus providing effectiveand optimized use of both the customer's PNAP and NSP pipes based on theASimilater information related to said customer over the PNAP BGP feed.

On the other hand, when the preferred link is over the PNAP 20 (e.g.,when destination 3 is not a customer of NSP D to which customer 1 isalso connected), the data packet is transmitted from customer 1 overlink 24 to the left half of the PNAP 20. The PNAP routing infrastructurewithin the PNAP 20 will have determined a plurality of paths todestination 3. These different paths to the same destination are listedin a routing table along with a parameter indicating the degree ofpreference attached to each route of a set of the different paths. Bymanipulating the local preference component of the route selectionprocess of the BGP4 protocol, the PNAP 20 picks the best path for thetraffic to traverse to reach destination 3. The data packet leaves theright side of the PNAP 20 via the selected one of the NSPs A-N, followsthe selected best path through the Internet 22, and reaches destination3.

Therefore, in accordance with the present invention, two customersconnected to the same PNAP 20 see the PNAP 20 as the best path, andexchange traffic with each other through the PNAP 20 without ever goingout over the backbones of the NSPs A-N. Or, if a PNAP customer isdirectly connected to a particular NSP to which a destination is alsoconnected, the PNAP customer can utilize that NSP connection to send thetraffic to the destination based on the ASimilater information receivedover the BGP peering with the PNAP.

In the case of sending information from customer 1 to customer 2, therouting table inside the PNAP 20 would list the direct connection fromcustomer 1 to customer 2 through the left half of the PNAP 20 over thedotted path 25 as the optimum route. This means that communicationsbetween customer 1 and customer 2 who are connected to the PNAP 20should always use the dotted path 25 as the preferred path unless afailure or flaw prevents that path from being used.

Thus far we have described what will be referred to as “generic”diversity+. When a PNAP customer is multi-homed to more than one PNAPand one NSP, routing outbound traffic become increasingly complex. Byway of additional background, the invention described in U.S. Pat. No.6,009,081 subscribes to the model of symmetrical routing of traffic.This method allows us to bypass the public NAPs for approximately ninetypercent of the traffic flowing in and out of our PNAPs with theassociated benefit of much higher performance than is normal experiencedin today's Internet.

The way we accomplish this symmetrical when optimal routing of trafficis by use of our routing technology called ASimilater. Each PNAP hasit's own BGP AS and is completely distinct from the routing perspectiveof the other PNAPs with no private backbone connecting the PNAPs.

Each PNAP is, however, connected to the same fabric of NSPs as all otherPNAPS. The levels of bandwidth to a PNAP may be larger or smallerdepending on it's location but the fabric is the same. With that inmind, let us examine the routing of PNAP-SFJ as an example.

First, assume that each PNAP is connected to the same fabric of NSPs asall other PNAPs. Generally speaking, routing of traffic inbound from anNSP over the pipe to said NSP is easy. All of these NSPs attach a higherlocal preference to the routes heard from their customers over thosesame routes heard from their peers. Routing outbound traffic in amassively multi-homed network is much more difficult. Faced with such amultiplicity of links, the question of how to route traffic in a tightlycontrolled fashion is one of great importance in attaining the highestperformance.

Note that we do not peer with the NSPs that we connect to but arerather, full transit customers of each one. This allows us to receiveeach NSPs perspective on the global routing table. ASimilater collatesthat data together and builds an interconnection matrix of the entireInternet. With that information, ASimilater can then route trafficoptimally from each PNAP.

An additional function of ASimilater is to control the interPNAProuting. We optimize the connectivity between the PNAPs as well since wecan use any of the NSPs connecting the PNAPs to route traffic betweenthem. This allows us to choose the fastest NSP between any two PNAPs,and thus allows us to offer the optimal path between our customers andthe Internet.

In the case of Diversity+, we offer our customers access to ASimilaterdata over their BGP feed to the PNAP 20 by use of the BGP communityattribute. In other words, if a customer is connected to NSP C and aPNAP, we can offer our customer all of NSP C and NSP C's customersroutes tagged with a specific community InterNAP community, in this case6993:XXX.

That information allows our customer to route traffic destined to NSP Cand NSP C customers over the NSP C link and all other traffic routedover the PNAP connection. This allows a customer to enjoy the sameperformance gains of symmetrical routing of traffic as PNAP even over apipe not connected to the PNAP 20.

Referring also to FIG. 4, in the customer 5 topology there is aconnection to NSP A, a connection to NSP B and a connection to InterNAP(PNAP-SFJ). In this topology, we recommend the following configuration:

-   -   (a) NSP A customer routes over the NSP A link.    -   (b) NSP B customer routes over the NSP B link.    -   (c) All others over the PNAP link.

In order for this to occur, we send the customer NSP A and NSP B routestagged with the following communities:

-   -   NSP A: 6993:NSP A    -   NSP B: 6993:NSP B

For clarity, let's create the table of the local pref values to use inour IBGP. TABLE 1 NSP B NSP A PNAP NSP B 80 45 75 NSP A 40 90 75 PNAP 4045 150

Setting the fall-through local pref values to half of the primaryassists in understanding from what peer a route is being heard whenperusing the BGP table. For example, in Table 1 all NSP A routes areassigned a local pref of 90 and all of the other routes heard from NSP Aare assigned a local pref of 45. If you were to see a route tagged at alocal pref of 45 in your IBGP, that would signify a non-NSP A routeannounced to the customer over the customer's BGP peering with NSP A.

The net effect of this local pref hierarchy is that of the routes thatwe know are not NSP B or NSP A, highest local pref wins on the PNAPlink. The fall-through local pref value is used in the case of multipleroutes heard over >1 of your connections. Multi-homed customers of thePNAP, NSP A, and NSP B would use the PNAP and, if that link was notavailable, the NSP A link followed by NSP B. Multi-homed customers ofNSP B and NSP A would, in the example above, use NSP A followed by NSPB.

Whether using NSP A or NSP B in the case of a multi-homed customer ofboth is entirely at the customer's discretion. That behavior is easilymodifiable by switching the primary and fall-through local pref sets ofNSP A and NSP B.

EXAMPLE 1

The following is an example of implementing this approach with NSP A.NSP A peer:

-   -   neighbor xxx.xxx.xxx.xxx remote-as    -   neighbor xxx.xxx.xxx.xxx send-community    -   neighbor xxx.xxx.xxx.xxx remote-as NSP A    -   neighbor xxx.xxx.xxx.xxx version 4    -   neighbor xxx.xxx.xxx.xxx distribute-list 1 out    -   neighbor xxx.xxx.xxx.xxx route-map NSP A-IN in    -   neighbor xxx.xxx.xxx.xxx route-map NSP A-OUT out    -   neighbor xxx.xxx.xxx.xxx filter-list 1 out    -   spr-bgw-02#    -   ip as-path access-list 1 permit {circumflex over ( )}$    -   ip as-path access-list 2 permit.*    -   ip as-path access-list 10 deny {circumflex over ( )}NSP A_NSP        B_.*    -   ip as-path access-list 10 deny {circumflex over ( )}NSP        A_XXXXX_.*    -   route-map NSP A-OUT permit 10    -   ! only allow customer 5 IBGP sourced routes    -   match as-path 1    -   route-map NSP A-IN permit 10    -   ! let's start by denying all routes we know are NSP B and        PNAP-SEA    -   ! and attaching a medium primary local pref.    -   match as-path 10    -   set local-preference 90    -   route-map NSP A-IN permit 20    -   ! Any other routes attach a medium fall through local pref match        as-path 2    -   set local-preference 45    -   Internap Router:    -   neighbor xxx.xxx.xxx.xxx remote-as XXXXX    -   neighbor xxx.xxx.xxx.xxx send-community    -   neighbor xxx.xxx.xxx.xxx version 4    -   neighbor xxx.xxx.xxx.xxx distribute-list 1 out    -   neighbor xxx.xxx.xxx.xxx route-map PNAP-IN in    -   neighbor xxx.xxx.xxx.xxx route-map PNAP-OUT out    -   neighbor xxx.xxx.xxx.xxx filter-list 1 out    -   ip community-list 1 deny 6993:NSP A; deny NSP A routes    -   ip community-list 1 deny 6993:NSP B; deny NSP B routes    -   ip as-path access-list 1 permit {circumflex over ( )}$    -   ip as-path access-list 2 permit.*    -   route-map PNAP-OUT permit 10    -   ! only allow customer 5 IBGP sourced routes    -   ! his is already being accomplished by the distribute-list    -   ! out but this routemap is where you can adjust your AS    -   ! prependings.    -   match as-path 1    -   route-map PNAP-IN permit 10    -   ! any routes that we know are not NSP B, or NSP A tag    -   highest    -   ! primary local pref    -   match community 1    -   set local-preference 150    -   route-map PNAP-IN permit 20    -   ! all else (NSP A, and NSP B routes) tag highest    -   fall through    -   ! local pref    -   ! all else (NSP A, and NSP B routes) tag highest fall    -   through    -   ! local pref    -   match as-path 2    -   set local-preference 75    -   NSP B Router:    -   neighbor 144.228.98.5 remote-as NSP B    -   neighbor 144.228.98.5 version 4    -   neighbor 144.228.98.5 distribute-list 1 out    -   neighbor 144.228.98.5 route-map NSP B-IN in    -   neighbor 144.228.98.5 route-map NSP B-OUT out    -   neighbor 144.228.98.5 filter-list 1 out    -   ip as-path access-list 1 permit {circumflex over ( )}$    -   ip as-path access-list 2 permit.*    -   ip as-path access-list 10 deny {circumflex over ( )}NSP        B_XXXXX_.*    -   ip as-path access-list 10 deny {circumflex over ( )}NSP B_NSP        A_.*    -   ip as-path access-list 10 deny {circumflex over ( )}NSP        B_(—)1664_.*    -   route-map NSP B-OUT permit 10    -   ! only allow customer 5 IBGP sourced routes    -   ! this is already being accomplished by the distribute-list    -   ! out but this routemap is where you can adjust your AS    -   ! prependings.    -   match as-path 1    -   route-map NSP B-IN permit 10    -   deny all NSP A, and PNAP routes and set a low    -   ! primary local pref    -   match as-path 10    -   set local-preference 80    -   !    -   route-map NSP B-IN permit 20    -   ! All else tag with a lowest fall through local pref.    -   match as-path 2    -   set local-preference 40

There is another configuration which also requires specialconsideration; namely, where a multi-homed PNAP customer with genericDiversity+ is connected to more than one PNAP.

The local-preference hierarchy of generic Diversity+ is intended toaddress the problem of multi-PNAP routing by creating an interlockingset of preference steps for path selection. In its defaultconfiguration, generic Diversity+ supports up to 2 PNAP transitconnections and multiple, other NSP transit connections.

Each primary level of local-preference has a corresponding secondaryvalue used as a backup should the primary become invalid. The completehierarchy is shown below.

Generic Diversity+Local Preference Hierarchy (Default)

400 PNAP Direct Customer High (Primary Link)

350 PNAP Direct Customer Low (Secondary Link)

300 Primary PNAP Direct NSP

250 Secondary PNAP Direct NSP

200 Primary PNAP Non-connected

150 Secondary PNAP Non-connected

100 Default Local Preference Value

90 Primary PNAP Direct NSP Backup

80 Secondary PNAP Direct NSP Backup

70 Primary PNAP Non-connected Backup

60 Secondary PNAP Non-connected Backup

This hierarchy is applied as follows:

For customers with no more than 1 link to a given PNAP, routes tocustomers of that PNAP are set to 400. When a customer has single linksto multiple PNAPs, the value is still set to 400 and the length of theAS path is left to break the tie, meaning the direct link to the PNAPsourcing those customer routes will be used as the AS path will beshorter.

If a customer has multiple links to the same PNAP, then routes over theprimary link to customers of that PNAP will be set to 400, while routesover the secondary link to those same customer routes will be set to350.

Routes belonging to NSPs and their customers directly connected to theprimary PNAP are set to 300, while routes belonging to NSPs and theircustomers directly connected to the secondary PNAP are set to 250. Thisresults in traffic being sent through the primary PNAP if the primaryPNAP has a given NSP in its border fabric. If the secondary PNAP has anNSP in its border fabric not common to the primary PNAP, or if an NSPcommon to them both fails at the primary, the traffic will be sentthrough the secondary for those destinations.

For destinations within NSPs which are not part of the border fabric ofthe primary PNAP routes are set to 200. Similar routes from thesecondary PNAP are set to 150.

Should an NSP connection at the primary PNAP fail, routes to that NSPthrough the primary PNAP will be set to 200, rather than 300. If an NSPconnection at the secondary PNAP fails, routes to that NSP through thesecondary PNAP will be set to 150, rather than 250.

The default value of 100 is generally not used for routes through a PNAPand is instead allocated for cases in which a customer has a connectionto another NSP in addition to a PNAP.

The values below 100 are used for customer NSP routes heard through thePNAP. The routes heard via the primary PNAP from the NSP to which thecustomer has a direct connection are set to 90. The same routes heardfrom the secondary PNAP are set to 80. Both of these cases assume thePNAPs have the NSP in their border fabric.

If the customer has a connection to an NSP not found in the borderfabric of the primary PNAP, those routes heard through the primary PNAPfor destinations within that NSP are set to 70. If such is the case withthe secondary PNAP, those routes are set to 60.

Determining Primary and Secondary

In a simple multi-PNAP scenario, a customer is connected to more thanone PNAP in a given city or region and the primary and secondary PNAPscan be determined based on traffic levels within the PNAPs, providerfabric, or other concerns. However, when the multiple PNAPs are not allgeographically close, a simple primary/secondary configuration mayresult in sub-optimal routing both in and out of the customer network.

In cases when a customer is connected to multiple, geographicallydiverse PNAPs the preferred configuration is to have multiple primaries,one per region. In this way, PNAP NSPs will use their IGP cost forinbound traffic and the customer can similarly use their own IGP costfor outbound traffic. Care must be taken to properly announce prefixesto control regional traffic flows. Customers with such disperse PNAPconnectivity should announce both their aggregate networks as well asmore specific, regional prefixes.

As an example, consider a customer with sites in both LAX and NYC withtheir own backbone connection between them. Each site connects to onePNAP in their area. The customer has been allocated 192.168.0.0/16 andhas internally allocated 192.168.0.0/17 for the LAX site and192.168.128.0/17 for the NYC site. From the LAX PNAP they would announceboth 192.168.0.0/16 and 192.168.0.0/17. From the NYC PNAP they wouldannounce both 192.168.0.0/16 and 192.168.128.0/17. If the customerwished to avoid any traffic to or from external destinations fromtransiting their backbone, they would instead advertise only the morespecific prefixes (192.168.0.0/17 and 192.168.128.0/17) and not theaggregate (192.168.0.0/16).

This multiple primary PNAP model can be extended to an arbitrary numberof regions, but within a single region, there must be a single primary.

EXAMPLE 2 Configuration for a Multi-PNAP Customer

In the example below assume the customer is connected to 2 PNAPs, A andB. A is the primary, with connections to NSP C, NSP D, while B is thesecondary, with connections to NSP C, NSP D, and NSP E.

PNAP Data for Customer Configuration: PNAP A Autonomous System Number:XXXXX Border 1 Next Hop: 10.8.230.1 Internal/Customer Networks:10.8.0.0/16 192.168.4.0/24 (AS 12005) 192.168.16.0/20 (AS 5507) NSPFabric: NSP D (AS 1239) NSP C (AS 701) PNAP B Autonomous System Number:6993 Border 2 Next Hop: 172.18.24.33 Internal/Customer Networks:172.18.0.0/16 172.20.4.0/22 (AS 13461) NSP Fabric: NSP D (AS 1239) NSP C(AS 701) NSP E (AS 3561)

EXAMPLE 3 BGP Routes Selected at Customer

Customer-CPE>sho ip bgp

BGP table version is 3063602, local router ID is 10.8.230.2

Status codes: s suppressed, d damped, h history, * valid, >best,i—internal

Origin codes: i—IGP, e—EGP, ?—incomplete Network Next Hop Metric LocPrfWeight Path *>i9.2.0.0/16 10.8.230.1 0 300 0 XXXXX XXX i * 172.18.24.330 250 0 6993 XXX i *>i10.8.0.0/16 10.8.230.1 0 400 0 XXXXX i *172.18.24.33 0 150 0 6993 1239 XXXXX i *  24.116.4.0/23 10.8.230.1 0 2000 XXXXX 1239 3561 i *>i 172.18.24.33 0 250 0 6993 3561 i *>i137.99.0.010.8.230.1 0 200 0 XXXXX 1239 209 i * 172.18.24.33 0 150 0 6993 1239 209i *  172.18.0.0 10.8.230.1 0 200 0 XXXXX XXX 6993 i *>i 172.18.24.33 0400 0 6993 i *  172.20.4.0/22 10.8.230.1 0 200 0 XXXXX XXX 6993 13461 i*>i 172.18.24.33 0 400 0 6993 13461 i *>i192.168.4.0 10.8.230.1 0 400 0XXXXX 12005 i * 172.18.24.33 0 150 0 6993 1239 XXXXX 12005 i*>i192.168.16.0/20 10.8.230.1 0 400 0 XXXXX 5507 i * 172.18.24.33 0 1500 6993 1239 XXXXX 5507 i

Detail of BGP Route Information for Specific Prefixes

Customer-CPE>sho ip bgp 10.8.0.0

BGP routing table entry for 10.8.0.0/16, version 1304669

Paths: (2 available, best #2)

-   -   6993 1239 XXXXX        -   172.18.24.33 from 172.18.24.33 (172.18.24.1)            -   Origin IGP, metric 0, localpref 150, valid, external    -   XXXXX        -   10.8.230.1 from 10.8.230.1 (10.8.230.1)            -   Origin IGP, metric 0, localpref 400, valid, external,                best

Customer-CPE>sho ip bgp 137.99.0.0

BGP routing table entry for 137.99.0.0, version 1304669

Paths: (2 available, best #2)

-   -   6993 1239 209        -   172.18.24.33 from 172.18.24.33 (172.18.24.1)            -   Origin IGP, metric 0, localpref 150, valid, external    -   XXXXX 1239 209        -   10.8.230.1 from 10.8.230.1 (10.8.230.1)            -   Origin IGP, metric 0, localpref 200, valid, external,                best

Although the description above contains many specificities, these shouldnot be construed as limiting the scope of the invention but as merelyproviding illustrations of some of the presently preferred embodimentsof this invention. Thus the scope of this invention should be determinedby the appended claims and their legal equivalents. Therefore, it willbe appreciated that the scope of the present invention fully encompassesother embodiments which may become obvious to those skilled in the art,and that the scope of the present invention is accordingly to be limitedby nothing other than the appended claims, in which reference to anelement in the singular is not intended to mean “one and only one”unless explicitly so stated, but rather “one or more.” All structural,chemical, and functional equivalents to the elements of theabove-described preferred embodiment that are known to those of ordinaryskill in the art are expressly incorporated herein by reference and areintended to be encompassed by the present claims. Moreover, it is notnecessary for a device or method to address each and every problemsought to be solved by the present invention, for it to be encompassedby the present claims. Furthermore, no element, component, or methodstep in the present disclosure is intended to be dedicated to the publicregardless of whether the element, component, or method step isexplicitly recited in the claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. 112, sixth paragraph, unlessthe element is expressly recited using the phrase “means for.”

1. A packet-switched network, comprising: a Private Network Access Point(PNAP) having a customer side and a service provider side; at least afirst customer and a second customer connected to the customer side ofthe PNAP; at least one service provider connected to the serviceprovider side of the PNAP; and an interconnected network systemaccessible to said service provider and to said first and secondcustomers; wherein traffic between said first and second customers isexchanged through the PNAP without transiting over said serviceprovider.
 2. A network as recited in claim 1, wherein the PNAP routinginfrastructure within said PNAP contains a routing table listing allroutes to all destinations in the network, along with a parameterindicating the degree of preference attached to each route of a set ofidentical routes from multiple sources.
 3. A network as recited in claim1, wherein said PNAP routing infrastructure lists a direct connectionbetween said first and second customers within said customer side of thePNAP, and wherein said PNAP routing infrastructure sets the level ofpreference higher for said direct connection than for any other routesbetween said first and second customers.
 4. A network as recited inclaim 1, wherein at least one of said customers is connected to aservice provider who is also connected to the service provider side ofthe PNAP so that said customer is multi-homed.
 5. A network as recitedin claim 4, wherein said multi-homed customer is provided a routingtable listing all routes to all destinations in the network, along witha community attribute denoting the preferred reachability of a givenroute over the customer non-PNAP connection.
 6. A network as recited inclaim 5, wherein said multi-homed customer can utilize the ASimilaterdata received over the BGP feed from the PNAP and send traffic to adestination served by said service provider through said customer'sconnection to said service provider by using said routing table to setroute preferences in a router maintained by said customer.
 7. Apacket-switched network comprising: a Private Network Access Point(PNAP) having a customer side and a service provider side; at least afirst customer and a second customer connected to the customer side ofthe PNAP; a plurality of service providers connected to the serviceprovider side of the PNAP; and an interconnected network systemaccessible to said plurality of service providers and to said first andsecond customers; wherein the PNAP routing infrastructure within saidPNAP contains a routing table listing all routes to all destinations inthe network, along with a parameter indicating the degree of preferenceattached to each route of a set of identical routes from multiplesources; wherein said PNAP routing infrastructure lists a directconnection between said first and second customers within said customerside of the PNAP.
 8. A network as recited in claim 7, wherein said PNAProuting infrastructure sets the level of preference higher for saiddirect connection than for any other routes between said first andsecond customers.
 9. A network as recited in claim 7, wherein trafficbetween said first and second customers is exchanged through the PNAPwithout transiting over said service provider.
 10. A network as recitedin claim 7, wherein at least one of said customers is connected to aservice provider who is also connected to the service provider side ofthe PNAP so that said customer is multi-homed.
 11. A network as recitedin claim 10, wherein said multi-homed customer is provided with saidrouting table listing all routes to all destinations in the network,along with a community attribute denoting the preferred reachability ofa given route over the customer non-PNAP connection.
 12. A network asrecited in claim 11, wherein said multi-homed customer can utilize theASimilater data received over the BGP feed from the PNAP and sendtraffic to a destination served by said service provider through saidcustomer's connection to said service provider by using said routingtable to set route preferences in a router maintained by said customer.13. A method for exchanging traffic in a packet-switched network,comprising: providing a Private Network Access Point (PNAP) having acustomer side and a service provider side; connecting at least a firstcustomer and a second customer to the customer side of the PNAP;connecting at least one service provider to the service provider side ofthe PNAP; making an interconnected network system accessible to saidservice provider and to said first and second customers; and causingtraffic between said first and second customers to be exchanged throughthe PNAP without transiting over said service provider.
 14. A network asrecited in claim 13, wherein the PNAP routing infrastructure within saidPNAP contains a routing table listing all routes to all destinations inthe network, along with a parameter indicating the degree of preferenceattached to each route of a set of identical routes from multiplesources.
 15. A network as recited in claim 1, wherein said PNAP routinginfrastructure lists a direct connection between said first and secondcustomers within said customer
 16. A method as recited in claim 15,wherein said PNAP routing infrastructure sets the level of preferencehigher for said direct connection than for any other routes between saidfirst and second customers.
 17. A network as recited in claim 13,wherein at least one of said customers is connected to a serviceprovider who is also connected to the service provider side of the PNAPso that said customer is multi-homed.
 18. A network as recited in claim17, further comprising providing said multi-homed customer with arouting table listing all routes to all destinations in the network,along with a community attribute denoting the preferred reachability ofa given route over the customer non-PNAP connection.
 19. A network asrecited in claim 18, further comprising allowing said multi-homedcustomer to utilize ASimilater data received over the BGP feed from thePNAP and send traffic to a destination served by said service providerthrough said customer's connection to said service provider by usingsaid routing table to set route preferences in a router maintained bysaid customer.
 20. A method for exchanging traffic in a packet-switchednetwork, comprising: providing a Private Network Access Point (PNAP)having a customer side and a service provider side; connecting at leasta first customer and a second customer to the customer side of the PNAP;connecting at least one service provider to the service provider side ofthe PNAP; and making an interconnected network system accessible to saidservice provider and to said first and second customers; wherein thePNAP routing infrastructure within said PNAP contains a routing tablelisting all routes to all destinations in the network, along with aparameter indicating the degree of preference attached to each route ofa set of identical routes from multiple sources; wherein said PNAProuting infrastructure lists a direct connection between said first andsecond customers within said customer side of the PNAP.
 21. A method asrecited in claim 20, further comprising causing traffic between saidfirst and second customers to be exchanged through the PNAP withouttransiting over said service provider.
 22. A method as recited in claim20, wherein said PNAP routing infrastructure sets the level ofpreference higher for said direct connection than for any other routesbetween said first and second customers.
 23. A method as recited inclaim 20, wherein at least one of said customers is connected to aservice provider who is also connected to the service provider side ofthe PNAP so that said customer is multi-homed.
 24. A method as recitedin claim 23, further comprising providing said multi-homed customer withsaid routing table listing all routes to all destinations in thenetwork, along with a community attribute denoting the preferredreachability of a given route over the customer non-PNAP connection. 25.A method as recited in claim 24, further comprising allowing saidmulti-homed customer to utilize ASimilater data received over the BGPfeed from the PNAP and send traffic to a destination served by saidservice provider through said customer's connection to said serviceprovider by using said routing table to set route preferences in arouter maintained by said customer.
 26. A packet-switched networkcomprising: a Private Network Access Point (PNAP) having a customer sideand a service provider side; at least one customer connected to thecustomer side of the PNAP; at least one service provider connected tothe service provider side of the PNAP; and an interconnected networksystem accessible to said service provider and to said customer; whereinsaid customer is connected to a service provider who is also connectedto the service provider side of the PNAP so that said customer ismulti-homed; wherein said multi-homed customer is provided a routingtable listing all routes to all destinations in the network, along witha community attribute denoting the preferred reachability of a givenroute over the customer non-PNAP connection; and wherein saidmulti-homed customer can utilize the ASimilator data received over theBGP feed from the PNAP and send traffic to a destination served by saidservice provider through said customer's connection to said serviceprovider by using said routing table to set route preferences in arouter maintained by said customer.
 27. A packet-switched method forexchanging traffic in a packet-switched network, comprising: providing aPrivate Network Access Point (PNAP) having a customer side and a serviceprovider side; connecting at least one customer to the customer side ofthe PNAP; connecting at least one service provider to the serviceprovider side of the PNAP; making an interconnected network systemaccessible to said service provider and to said customer; wherein saidcustomer is connected to a service provider who is also connected to theservice provider side of the PNAP so that said customer is multi-homed;providing said multi-homed customer a routing table listing all routesto all destinations in the network, along with a community attributedenoting the preferred reachability of a given route over the customernon-PNAP connection; and allowing said multi-homed customer to utilizethe ASimilator data received over the BGP feed from the PNAP and sendtraffic to a destination served by said service provider through saidcustomer's connection to said service provider by using said routingtable to set route preferences in a router maintained by said customer.